Security News
Python Overtakes JavaScript as Top Programming Language on GitHub
Python becomes GitHub's top language in 2024, driven by AI and data science projects, while AI-powered security tools are gaining adoption.
sentence-splitter
Advanced tools
Split {Japanese, English} text into sentences.
This library split next text into 3 sentences.
We are talking about pens.
He said "This is a pen. I like it".
I could relate to that statement.
Result is:
You can check actual AST in online playground.
Second sentence includes "This is a pen. I like it"
, but this library can not split it into new sentence.
The reason is "..."
and 「...」
text is ambiguous as a sentence or a proper noun.
Also, HTML does not have suitable semantics for conversation.
As a result, The second line will be one sentence, but sentence-splitter add a contexts
info to the sentence node.
{
"type": "Sentence",
"children": [
{
"type": "Str",
"value": "He said \"This is a pen. I like it\""
},
...
],
"contexts": [
{
"type": "PairMark",
"pairMark": {
"key": "double quote",
"start": "\"",
"end": "\""
},
"range": [
8,
33
],
...
]
]
}
Probably, textlint rule should handle the "..."
and 「...」
text after parsing sentences by sentence-splitter.
npm install sentence-splitter
export interface SeparatorParserOptions {
/**
* Recognize each characters as separator
* Example [".", "!", "?"]
*/
separatorCharacters?: string[]
}
export interface AbbrMarkerOptions {
language?: Language;
}
export interface splitOptions {
/**
* Separator & AbbrMarker options
*/
SeparatorParser?: SeparatorParserOptions;
AbbrMarker?: AbbrMarkerOptions;
}
/**
* split `text` into Sentence nodes.
* This function return array of Sentence nodes.
*/
export declare function split(text: string, options?: splitOptions): TxtParentNodeWithSentenceNode["children"];
/**
* Convert Paragraph Node to Paragraph Node that includes Sentence Node.
* Paragraph Node is defined in textlint's TxtAST.
* See https://github.com/textlint/textlint/blob/master/docs/txtnode.md
*/
export declare function splitAST(paragraphNode: TxtParentNode, options?: splitOptions): TxtParentNodeWithSentenceNode;
See also TxtAST.
This node is based on TxtAST.
Str
: Str node has value
. It is same as TxtAST's Str
node.Sentence
: Sentence Node has Str
, WhiteSpace
, or Punctuation
nodes as childrenWhiteSpace
: WhiteSpace Node has \n
.Punctuation
: Punctuation Node has .
, 。
Get these SentenceSplitterSyntax
constants value from the module:
import { SentenceSplitterSyntax } from "sentence-splitter";
console.log(SentenceSplitterSyntax.Sentence);// "Sentence"
export type SentencePairMarkContext = {
type: "PairMark";
range: readonly [startIndex: number, endIndex: number];
loc: {
start: {
line: number;
column: number;
};
end: {
line: number;
column: number;
};
};
};
export type TxtSentenceNode = Omit<TxtParentNode, "type"> & {
readonly type: "Sentence";
readonly contexts?: TxtPairMarkNode[];
};
export type TxtWhiteSpaceNode = Omit<TxtTextNode, "type"> & {
readonly type: "WhiteSpace";
};
export type TxtPunctuationNode = Omit<TxtTextNode, "type"> & {
readonly type: "Punctuation";
};
Fore more details, Please see TxtAST.
Node layout image.
This is 1st sentence. This is 2nd sentence.
<Sentence>
<Str /> |This is 1st sentence|
<Punctuation /> |.|
</Sentence>
<WhiteSpace /> | |
<Sentence>
<Str /> |This is 2nd sentence|
<Punctuation /> |.|
</Sentence>
Note: This library will not split Str
into Str
and WhiteSpace
(tokenize)
Because, Tokenize need to implement language specific context.
You can use splitAST
for textlint rule.
splitAST
function can preserve original AST's position unlike split
function.
import { splitAST, SentenceSplitterSyntax } from "sentence-splitter";
export default function(context, options = {}) {
const { Syntax, RuleError, report, getSource } = context;
return {
[Syntax.Paragraph](node) {
const parsedNode = splitAST(node);
const sentenceNodes = parsedNode.children.filter(childNode => childNode.type === SentenceSplitterSyntax.Sentence);
console.log(sentenceNodes); // => Sentence nodes
}
}
}
Examples
This library use "Golden Rule" test of pragmatic_segmenter
for testing.
Run tests:
npm test
Create input.json
from _input.md
npm run createInputJson
Update snapshots(output.json
):
npm run updateSnapshot
test/fixtures/<test-case-name>/
directorytest/fixtures/<test-case-name>/_input.md
with testing contentnpm run updateSnapshot
test/fixtures/<test-case-name>/output.json
git checkout -b my-new-feature
git commit -am 'Add some feature'
git push origin my-new-feature
MIT
FAQs
split {japanese, english} text into sentences.
The npm package sentence-splitter receives a total of 41,042 weekly downloads. As such, sentence-splitter popularity was classified as popular.
We found that sentence-splitter demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Python becomes GitHub's top language in 2024, driven by AI and data science projects, while AI-powered security tools are gaining adoption.
Security News
Dutch National Police and FBI dismantle Redline and Meta infostealer malware-as-a-service operations in Operation Magnus, seizing servers and source code.
Research
Security News
Socket is tracking a new trend where malicious actors are now exploiting the popularity of LLM research to spread malware through seemingly useful open source packages.